热门标签 | HotTags
当前位置:  开发笔记 > 编程语言 > 正文

时效|间接性_猿创征文|技术成长之路Java编程系列之文件OSS存储实践:AmazonS3实现文件上传下载

篇首语:本文由编程笔记#小编为大家整理,主要介绍了猿创征文|技术成长之路-Java编程系列之文件OSS存储实践:AmazonS3实现文件上传下载相关的知识,希望对你有一定的参考价值。

篇首语:本文由编程笔记#小编为大家整理,主要介绍了猿创征文|技术成长之路-Java编程系列之文件OSS存储实践:Amazon S3实现文件上传下载相关的知识,希望对你有一定的参考价值。



热门系列:

【Java编程系列】Minio实现文件上传下载




1、前言

        最近有一客户,需要独立部署文件存储到自己的私有化服务器,于是选择了让我们使用在AWS服务器的s3存储桶方式来实现。现做一下实践后的开发经验记录~~~




2、Amazon S3实战代码

2.1 S3的Sdk对接实现


2.1.1 S3 Maven依赖

        这里需要说明一下,目前封装好的java可用的sdk有两种实现依赖,第一种:


com.amazonaws
aws-java-sdk-s3
1.12.276

        以上是第一种封装依赖,但是这里面会有部分依赖缺失,在文末的总结中,可以看到完整的依赖方式。

        第二种封装依赖:



software.amazon.awssdk
s3
2.17.100


software.amazon.awssdk
kms
2.17.100


software.amazon.awssdk
s3control
2.17.100

        本来是使用第一种依赖来做的开发编码。后来因为一个403权限拒绝原因,一开始不确定是否是api问题,所以后来又使用了s3官方文档推荐的第二种依赖完成了开发实现,因此以下示例代码,全都是引用第二种maven依赖来完成的!!!



2.1.2 s3 client配置与初始化

        在开始实践编码前,先列一下本次开发中,我参考的一些api文档,希望能给有需要的朋友一些参考和帮助。

        S3官方开发指南文档:使用 AWS SDK for Java 的 Amazon S3示例 - AWS SDK for Java

        开发示例代码github地址:aws-doc-sdk-examples/javav2/example_code/s3/src/main/java/com/example/s3 at main · awsdocs/aws-doc-sdk-examples · GitHub

        还有第一种依赖的实现参考代码:如何使用AWS SDK for Java操作MinIO Server | Minio中文文档        

        s3存储桶的配置信息,我这边是使用数据表存储配置的,因为我们系统的多元化接入。可能会有多套配置,所以请按需配置。

        初始化代码如下,PrivateUploadConfig类是我的s3配置信息实体类:

import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.core.waiters.WaiterResponse;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadBucketRequest;
import software.amazon.awssdk.services.s3.model.HeadBucketResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
import software.amazon.awssdk.services.s3.waiters.S3Waiter;
import javax.annotation.Resource;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.time.Duration;
import java.util.Objects;
/**
* @Date: 2022/8/19 19:40
* @Description
*/
@Component
@Slf4j
public class S3ClientUtil
private S3Client s3Client;
// private S3AsyncClient s3AsyncClient;
@Resource(name = CommonBeanNames.STRING_REDIS_HELPER)
private RedisHelper redisHelper;

/**
* @Description
* @Date 19:53 2022/8/19
* @Param [uploadConfig=s3配置信息实体类]
* @return software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain
**/
private AwsCredentialsProviderChain getProvider(PrivateUploadConfig uploadConfig)
return AwsCredentialsProviderChain
.builder()
.addCredentialsProvider(new AwsCredentialsProvider()
@Override
public AwsCredentials resolveCredentials()
return AwsBasicCredentials.create(uploadConfig.getAccessKey(), uploadConfig.getSecretKey());

).build();


/**
* @return void
* @Description
* @Date 19:47 2022/8/19
* @Param [uploadConfig=s3配置信息实体类]
**/
private void init(PrivateUploadConfig uploadConfig)
if (null != s3Client)
return;

s3Client = S3Client.builder()
.credentialsProvider(getProvider(uploadConfig))
.region(Region.of(uploadConfig.getRegion()))
.build();


/**
* @Description
* @Date 19:53 2022/8/19
* @Param [bucket=存储桶名称]
* @return void
**/
public void makeBucket(String bucket)
S3Waiter s3Waiter = s3Client.waiter();
CreateBucketRequest bucketRequest = CreateBucketRequest.builder()
.bucket(bucket)
.build();
s3Client.createBucket(bucketRequest);
HeadBucketRequest bucketRequestWait = HeadBucketRequest.builder()
.bucket(bucket)
.build();
// Wait until the bucket is created and print out the response.
WaiterResponse waiterResponse = s3Waiter.waitUntilBucketExists(bucketRequestWait);
waiterResponse.matched().response().ifPresent(r->log.info(r.toString()));



2.1.3 上传文件

import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.core.waiters.WaiterResponse;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.CreateBucketRequest;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.HeadBucketRequest;
import software.amazon.awssdk.services.s3.model.HeadBucketResponse;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
import software.amazon.awssdk.services.s3.waiters.S3Waiter;
import javax.annotation.Resource;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.time.Duration;
import java.util.Objects;
/**
* @Date: 2022/8/19 19:40
* @Description
*/
@Component
@Slf4j
public class S3ClientUtil
private S3Client s3Client;
// private S3AsyncClient s3AsyncClient;
@Resource(name = CommonBeanNames.STRING_REDIS_HELPER)
private RedisHelper redisHelper;
public void uploadFile(PrivateUploadConfig uploadConfig, String fileName, MultipartFile multipartFile, String contentType)
throws IOException
if (null == s3Client)
init(uploadConfig);

PutObjectRequest.Builder putOb = PutObjectRequest.builder();
putOb.bucket(uploadConfig.getBucket());
putOb.key(fileName);
if (StringUtils.isNotEmpty(contentType))
putOb.contentType(contentType);

long begin = System.currentTimeMillis();
s3Client.putObject(putOb.build(), RequestBody.fromInputStream(multipartFile.getInputStream(), multipartFile.getInputStream().available()));
log.info("S3 putObject time :", System.currentTimeMillis() - begin);



2.1.4 下载文件

import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
import javax.annotation.Resource;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.time.Duration;
import java.util.Objects;
/**
* @Date: 2022/8/19 19:40
* @Description
*/
@Component
@Slf4j
public class S3ClientUtil
private S3Client s3Client;
// private S3AsyncClient s3AsyncClient;
private InputStream amazonS3Download(PrivateUploadConfig uploadConfig, String fileName)
if (null == s3Client)
init(uploadConfig);

//此处需要注意,如果fileName第一位有/,则需要剔除掉,否则会报错找不到
if (fileName.indexOf("/") == 0)
fileName = fileName.replaceFirst("/", "");

GetObjectRequest objectRequest = GetObjectRequest
.builder()
.key(fileName)
.bucket(uploadConfig.getBucket())
.build();
long begin = System.currentTimeMillis();
InputStream is = s3Client.getObject(objectRequest);
log.info("S3 getObject time :", System.currentTimeMillis() - begin);
return is;
// byte [] bytes = s3AsyncClient.getObject(objectRequest,AsyncResponseTransformer.toBytes()).thenApply(BytesWrapper::asByteArray).join();
// return new ByteArrayInputStream(bytes);



2.1.5 生成文件预签名url

import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentials;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
import software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain;
import software.amazon.awssdk.core.sync.RequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.GetObjectRequest;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;
import software.amazon.awssdk.services.s3.presigner.S3Presigner;
import software.amazon.awssdk.services.s3.presigner.model.GetObjectPresignRequest;
import software.amazon.awssdk.services.s3.presigner.model.PresignedGetObjectRequest;
import javax.annotation.Resource;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.time.Duration;
import java.util.Objects;
/**
* @Date: 2022/8/19 19:40
* @Description
*/
@Component
@Slf4j
public class S3ClientUtil
private S3Client s3Client;
// private S3AsyncClient s3AsyncClient;
@Resource(name = CommonBeanNames.STRING_REDIS_HELPER)
private RedisHelper redisHelper;
public String getObjectURL(PrivateUploadConfig uploadConfig, String fileName)
if (fileName.indexOf("/") == 0)
fileName = fileName.replaceFirst("/", "");

S3Presigner presigner = S3Presigner.builder()
.credentialsProvider(getProvider(uploadConfig))
.region(Region.of(uploadConfig.getRegion()))
.build();
GetObjectRequest getObjectRequest = GetObjectRequest.builder()
.bucket(uploadConfig.getBucket())
.key(fileName)
.build();
//默认生成的预览url最大时效时间为:7天,aws s3能允许的最大时限也是7天
GetObjectPresignRequest getObjectPresignRequest = GetObjectPresignRequest.builder()
.signatureDuration(Duration.ofDays(7))
.getObjectRequest(getObjectRequest)
.build();
PresignedGetObjectRequest presignedGetObjectRequest = presigner.presignGetObject(getObjectPresignRequest);
String theUrl = presignedGetObjectRequest.url().toString();
log.info("Origin path : , Presigned URL: ", fileName, theUrl);
return theUrl;



2.1.6 文件“永久性”访问实现

        其实S3存储桶,通过策略配置,是可以直接通过 域名+桶名称+文件路径名称 ,来直接永久访问到的。但是,这么做的话,安全性就大大降低了。所以,我们需要采取其他的方式来间接实现永久访问的功能。于是,有了一下曲线救国的方针:



        通过定时任务,根据你生成预签名url的最大时效来,往后推一点时间,来生成需要更新的时间周期。


        例如,我选择最大时效为7天,那么我会定时任务,每天查询图片url时效已达过6天的记录,然后重新生成新的url更新到记录里,如此,就可以间接性的永久有效啦~~~




2.1.7 其他操作(解决下载慢)

        还有很多其他api,如删除文件、获取存储桶列表、获取文件列表;另外还有一些,针对不同region或是高版本api,还有transfer等高效高性能方法可用。

        另外,使用第一种依赖,还能实现一些分段上传、下载。官方建议,当单个文件大小超过100M时,则需要使用分段上传下载处理。

        但是,我这里遇到了一个其他问题:上传和下载很慢的问题 ,这个问题我分析主要是来自于网络传输效率低所导致,不在代码层面。测试用例:一个10M左右的文件,上传需要10+秒,下载更是达到60秒左右。

        得出以上结论的原因有:



1、同样的s3实现,我们其他使用aws服务器的朋友,他们使用上传下载时,就没有这个慢的情况出现,至少不会像我这个情况。


2、我使用本地文件,对代码进行测试,除了s3的api调用处,其他代码运行效率都很高,只在毫秒级别,使用的是同一个大小的原文件测试的。


3、使用断点调试时,发现s3Client.putObject()方法,阻塞所占时间为主要耗时;另外,对于s3Client.getObject()下载方法,虽然程序调用在毫秒级,但最终将文件流输出时,底层实际是通过s3服务端的url实时传输的,所以也是传出效率低所导致!


        最终,我先想了一个方式,解决下载慢的问题。因为一个文件下载要1min多,实在是太不能接受了。上传10s左右,还得过且过吧。

        对于下载慢的问题,我使用的方式时,在上传完成后,使用redis缓存文件流来解决!

String fis = new String(multipartFile.getBytes(),"ISO8859-1");
redisHelper.hset(cacheKey,fileId,fis);

        以上是缓存实现的关键代码。对于为什么字符串编码选择使用 ISO8859-1 ,因为也有与base64转码做过性能对比,发现前者的性能更优。大家可以参考使用!




3、总结

         整体实践完成后,有一些开发过程中遇到的问题,在这里总结一下。帮助大家避坑~~~


3.1 异常报错


错误一:关键依赖类缺失(EndpointToRegion类找不到)

        运行时异常主要信息为:



Caused by: java.lang.ClassNotFoundException: com.amazonaws.regions.EndpointToRegion


        全部报错信息如下:

org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: com/amazonaws/regions/EndpointToRegion
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1055)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ***.GzipFilter.doFilter(GzipFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ***.AbstractCorsFilter.doFilter(AbstractCorsFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ***.ChannelFilter.doFilter(ChannelFilter.java:33)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: com/amazonaws/regions/EndpointToRegion
at com.amazonaws.services.s3.internal.auth.S3SignerProvider.getSigner(S3SignerProvider.java:62)
at com.amazonaws.http.ExecutionContext.getSigner(ExecutionContext.java:135)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor$ExecOneRequestParams.newSigner(AmazonHttpClient.java:1843)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1216)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1074)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
at com.amazonaws.services.s3.AmazonS3Client.getAcl(AmazonS3Client.java:4062)
at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1278)
at com.amazonaws.services.s3.AmazonS3Client.getBucketAcl(AmazonS3Client.java:1268)
at com.amazonaws.services.s3.AmazonS3Client.doesBucketExistV2(AmazonS3Client.java:1406)
at ***.***.PrivateMinioClientHelper.makeBucket(PrivateMinioClientHelper.java:56)
at ***.***.PrivateMinioClientHelper.amazonS3Init(PrivateMinioClientHelper.java:140)
at ***.***.PrivateMinioClientHelper.init(PrivateMinioClientHelper.java:110)
at ***.***.PrivateMinioClientHelper.amazonS3Upload(PrivateMinioClientHelper.java:163)
at ***.***.PrivateMinioClientHelper.uploadFile(PrivateMinioClientHelper.java:72)
at ***.***.service.impl.PrivateMinioserviceImpl.uploadFile(PrivateMinioServiceImpl.java:79)
at ***.***.service.impl.AbstractFileServiceImpl.upload(AbstractFileServiceImpl.java:269)
at ***.***.controller.FileController.upload(FileController.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
... 52 common frames omitted
Caused by: java.lang.ClassNotFoundException: com.amazonaws.regions.EndpointToRegion
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 88 common frames omitted

解决方式:因为缺少部分依赖,只需要引入以下maven依赖即可


com.amazonaws
aws-java-sdk-s3
1.12.276


com.amazonaws
aws-java-sdk-kms
1.12.276



com.amazonaws
aws-java-sdk-core
1.12.276



com.amazonaws
jmespath-java
1.12.276


错误二:权限配置问题,拒绝访问

        另一个运行异常报错信息如下:



com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 7GR03AT1BWSMPNYM; S3 Extended Request ID: 8fu4lS0r0N6hNhKQbqunAnkmx3HxxRwjqA9oHobv61P7U8vYra23L4MuNlfC6sKjdFr73U1EmX4=; Proxy: null)


        整体错误日志如下:

com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 7GR03AT1BWSMPNYM; S3 Extended Request ID: 8fu4lS0r0N6hNhKQbqunAnkmx3HxxRwjqA9oHobv61P7U8vYra23L4MuNlfC6sKjdFr73U1EmX4=; Proxy: null)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1753)
at ***.helper.PrivateMinioClientHelper.amazonS3Upload(PrivateMinioClientHelper.java:174)
at ***.helper.PrivateMinioClientHelper.uploadFile(PrivateMinioClientHelper.java:73)
at ***.service.impl.PrivateMinioServiceImpl.uploadFile(PrivateMinioServiceImpl.java:79)
at ***.service.impl.AbstractFileServiceImpl.upload(AbstractFileServiceImpl.java:269)
at ***.controller.FileController.upload(FileController.java:108)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:879)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ***.GzipFilter.doFilter(GzipFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ***.AbstractCorsFilter.doFilter(AbstractCorsFilter.java:63)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at ***.ChannelFilter.doFilter(ChannelFilter.java:33)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:109)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:367)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1639)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
2022-08-10 10:37:52.535 [5m7a7kaVC5xwggeT] ERROR (GlobalExceptionHandler.java:135) - custom exception: code: 30006, message: null, data:null, Exception: ***.service.impl.PrivateMinioServiceImpl.uploadFile(PrivateMinioServiceImpl.java:89)
***.service.impl.AbstractFileServiceImpl.upload(AbstractFileServiceImpl.java:269)
***.controller.FileController.upload(FileController.java:108)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)

这个问题,提示已经很明显了,就是s3服务端的拒绝通过啦。当时,还一阵以为少了什么配置所致,其实不然,就是权限配置的问题。

解决方式:1、将IAM权限设置好,对指定账户放开哪些权限,如读写等等

                  2、将指定存储桶的权限放开,保证该桶对指定用户可以上传、下载、删除等等

         分享内容就到这里啦,欢迎大家留言探讨。有问题,也会第一时间回复~~


推荐阅读
  • 本文介绍了Python高级网络编程及TCP/IP协议簇的OSI七层模型。首先简单介绍了七层模型的各层及其封装解封装过程。然后讨论了程序开发中涉及到的网络通信内容,主要包括TCP协议、UDP协议和IPV4协议。最后还介绍了socket编程、聊天socket实现、远程执行命令、上传文件、socketserver及其源码分析等相关内容。 ... [详细]
  • 本文介绍了如何使用php限制数据库插入的条数并显示每次插入数据库之间的数据数目,以及避免重复提交的方法。同时还介绍了如何限制某一个数据库用户的并发连接数,以及设置数据库的连接数和连接超时时间的方法。最后提供了一些关于浏览器在线用户数和数据库连接数量比例的参考值。 ... [详细]
  • [译]技术公司十年经验的职场生涯回顾
    本文是一位在技术公司工作十年的职场人士对自己职业生涯的总结回顾。她的职业规划与众不同,令人深思又有趣。其中涉及到的内容有机器学习、创新创业以及引用了女性主义者在TED演讲中的部分讲义。文章表达了对职业生涯的愿望和希望,认为人类有能力不断改善自己。 ... [详细]
  • 本文介绍了Web学习历程记录中关于Tomcat的基本概念和配置。首先解释了Web静态Web资源和动态Web资源的概念,以及C/S架构和B/S架构的区别。然后介绍了常见的Web服务器,包括Weblogic、WebSphere和Tomcat。接着详细讲解了Tomcat的虚拟主机、web应用和虚拟路径映射的概念和配置过程。最后简要介绍了http协议的作用。本文内容详实,适合初学者了解Tomcat的基础知识。 ... [详细]
  • 本文介绍了计算机网络的定义和通信流程,包括客户端编译文件、二进制转换、三层路由设备等。同时,还介绍了计算机网络中常用的关键词,如MAC地址和IP地址。 ... [详细]
  • 利用Visual Basic开发SAP接口程序初探的方法与原理
    本文介绍了利用Visual Basic开发SAP接口程序的方法与原理,以及SAP R/3系统的特点和二次开发平台ABAP的使用。通过程序接口自动读取SAP R/3的数据表或视图,在外部进行处理和利用水晶报表等工具生成符合中国人习惯的报表样式。具体介绍了RFC调用的原理和模型,并强调本文主要不讨论SAP R/3函数的开发,而是针对使用SAP的公司的非ABAP开发人员提供了初步的接口程序开发指导。 ... [详细]
  • 如何实现织梦DedeCms全站伪静态
    本文介绍了如何通过修改织梦DedeCms源代码来实现全站伪静态,以提高管理和SEO效果。全站伪静态可以避免重复URL的问题,同时通过使用mod_rewrite伪静态模块和.htaccess正则表达式,可以更好地适应搜索引擎的需求。文章还提到了一些相关的技术和工具,如Ubuntu、qt编程、tomcat端口、爬虫、php request根目录等。 ... [详细]
  • Nginx使用AWStats日志分析的步骤及注意事项
    本文介绍了在Centos7操作系统上使用Nginx和AWStats进行日志分析的步骤和注意事项。通过AWStats可以统计网站的访问量、IP地址、操作系统、浏览器等信息,并提供精确到每月、每日、每小时的数据。在部署AWStats之前需要确认服务器上已经安装了Perl环境,并进行DNS解析。 ... [详细]
  • 本文详细介绍了SQL日志收缩的方法,包括截断日志和删除不需要的旧日志记录。通过备份日志和使用DBCC SHRINKFILE命令可以实现日志的收缩。同时,还介绍了截断日志的原理和注意事项,包括不能截断事务日志的活动部分和MinLSN的确定方法。通过本文的方法,可以有效减小逻辑日志的大小,提高数据库的性能。 ... [详细]
  • 搭建Windows Server 2012 R2 IIS8.5+PHP(FastCGI)+MySQL环境的详细步骤
    本文详细介绍了搭建Windows Server 2012 R2 IIS8.5+PHP(FastCGI)+MySQL环境的步骤,包括环境说明、相关软件下载的地址以及所需的插件下载地址。 ... [详细]
  • 阿,里,云,物,联网,net,core,客户端,czgl,aliiotclient, ... [详细]
  • 本文介绍了使用PHP实现断点续传乱序合并文件的方法和源码。由于网络原因,文件需要分割成多个部分发送,因此无法按顺序接收。文章中提供了merge2.php的源码,通过使用shuffle函数打乱文件读取顺序,实现了乱序合并文件的功能。同时,还介绍了filesize、glob、unlink、fopen等相关函数的使用。阅读本文可以了解如何使用PHP实现断点续传乱序合并文件的具体步骤。 ... [详细]
  • 本文介绍了Hyperledger Fabric外部链码构建与运行的相关知识,包括在Hyperledger Fabric 2.0版本之前链码构建和运行的困难性,外部构建模式的实现原理以及外部构建和运行API的使用方法。通过本文的介绍,读者可以了解到如何利用外部构建和运行的方式来实现链码的构建和运行,并且不再受限于特定的语言和部署环境。 ... [详细]
  • 本文介绍了通过ABAP开发往外网发邮件的需求,并提供了配置和代码整理的资料。其中包括了配置SAP邮件服务器的步骤和ABAP写发送邮件代码的过程。通过RZ10配置参数和icm/server_port_1的设定,可以实现向Sap User和外部邮件发送邮件的功能。希望对需要的开发人员有帮助。摘要长度:184字。 ... [详细]
  • 本文讨论了在VMWARE5.1的虚拟服务器Windows Server 2008R2上安装oracle 10g客户端时出现的问题,并提供了解决方法。错误日志显示了异常访问违例,通过分析日志中的问题帧,找到了解决问题的线索。文章详细介绍了解决方法,帮助读者顺利安装oracle 10g客户端。 ... [详细]
author-avatar
迪迪
这个家伙很懒,什么也没留下!
PHP1.CN | 中国最专业的PHP中文社区 | DevBox开发工具箱 | json解析格式化 |PHP资讯 | PHP教程 | 数据库技术 | 服务器技术 | 前端开发技术 | PHP框架 | 开发工具 | 在线工具
Copyright © 1998 - 2020 PHP1.CN. All Rights Reserved | 京公网安备 11010802041100号 | 京ICP备19059560号-4 | PHP1.CN 第一PHP社区 版权所有